Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add mapping tester reference results #212

Merged
merged 7 commits into from
Jan 28, 2025
Merged

Add mapping tester reference results #212

merged 7 commits into from
Jan 28, 2025

Conversation

fsimonis
Copy link
Member

Main changes of this PR

This PR adds a mapping tester compare script and reference results.

The script checks:

  • Existing columns
  • Equal test cases
  • Valid time and memory data (>0) as comparing is difficult across machines
  • Approx identical mapping results

Closes #208

Author's checklist

  • I used the pre-commit hook and used pre-commit run --all to apply all available hooks.
  • I added a test to cover the proposed changes in our test suite.
  • I updated the documentation in docs/README.md.
  • I added a changelog entry in ./changelog-entries/ (create if necessary).
  • I updated potential breaking changes in the tutorial precice/tutorials/aste-turbine.

@fsimonis fsimonis requested a review from davidscn October 25, 2024 15:26
davidscn
davidscn reviewed Oct 28, 2024
View reviewed changes
Copy link
Member

@davidscn davidscn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a great step for more robust testing. Currently, tests are failing.

tools/mapping-tester/compare.py Show resolved Hide resolved
@fsimonis fsimonis force-pushed the add-mapping-tester-compare-script branch from ac9c3ca to 91f154d Compare October 28, 2024 15:58
@fsimonis
Copy link
Member Author

I rewrote the compare file to be actually helpful now.

@fsimonis fsimonis force-pushed the add-mapping-tester-compare-script branch from 507e94d to 8898237 Compare January 23, 2025 10:43
@fsimonis
Copy link
Member Author

fsimonis commented Jan 23, 2025 •
edited
Loading

The results are still strange.

When running ctest -R mapping-tester locally with preCICE develop, I get for the NP mapping:

With clang (both) release

[preCICE:B:0] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.014321 avg: 0.000882676 var: 2.58172e-06 cnt: 1729
[preCICE:B:1] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.0137388 avg: 0.000680923 var: 2.41566e-06 cnt: 1729

With gcc (preCICE) release I get

[preCICE:B:0] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.014321 avg: 0.000882676 var: 2.58172e-06 cnt: 1729
[preCICE:B:1] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.0137388 avg: 0.000680923 var: 2.41566e-06 cnt: 1729

With gcc (both) release I get

[preCICE:B:0] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.014321 avg: 0.000882676 var: 2.58172e-06 cnt: 1729
[preCICE:B:1] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.0137388 avg: 0.000680923 var: 2.41566e-06 cnt: 1729

In the CI, we get

Clang debug

[preCICE:B:0] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.0132921 avg: 0.00088252 var: 2.69063e-06 cnt: 1743
[preCICE:B:1] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.0132319 avg: 0.000699396 var: 2.59288e-06 cnt: 1715

Clang release

[preCICE:B:0] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.0132921 avg: 0.00088171 var: 2.69091e-06 cnt: 1743
[preCICE:B:1] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.0132319 avg: 0.000701228 var: 2.61619e-06 cnt: 1715

GCC Debug

[preCICE:B:0] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.0132921 avg: 0.00088171 var: 2.69091e-06 cnt: 1743
[preCICE:B:1] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.0132319 avg: 0.000701228 var: 2.61619e-06 cnt: 1715

GCC Release

[preCICE:B:0] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.0132921 avg: 0.00088171 var: 2.69091e-06 cnt: 1743
[preCICE:B:1] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.0132319 avg: 0.000701228 var: 2.61619e-06 cnt: 1715

So, there seems to be enough variation in the nearest-projection to result in failing tests.

The big question is why are these projection results different.
Also, why are there still different amount of vertices in the mappings. Is the partitioning still off?

Turned out that the Metis partitioning is also non-deterministic.
Even if setting the seed manually, different versions return different results. Given that NP is hypersensitive to the partitioning, we decided to remove it from the parallel tests and use something else instead.

@fsimonis fsimonis force-pushed the add-mapping-tester-compare-script branch from 5080e84 to a70c9cd Compare January 23, 2025 14:42
@fsimonis fsimonis requested a review from davidscn January 23, 2025 14:42
@fsimonis fsimonis self-assigned this Jan 23, 2025
@fsimonis fsimonis added dependencies Pull requests that update a dependency file documentation Changes affecting the project's documentation labels Jan 23, 2025
@fsimonis fsimonis mentioned this pull request Jan 23, 2025
Open
@fsimonis fsimonis force-pushed the add-mapping-tester-compare-script branch from a70c9cd to 48df5db Compare January 23, 2025 14:51
@fsimonis
Copy link
Member Author

The NP mappings are indeed different between versions 1.86 and 1.87 of boost.

@davidscn
Copy link
Member

The NP mappings are indeed different between versions 1.86 and 1.87 of boost.

The solution should be unique, or am I missing something? This sounds like a bug either in preCICE or boost geometry. The release notes of boost 1.87 do not mention anything which could sound related.

@fsimonis
Copy link
Member Author

fsimonis commented Jan 24, 2025 •
edited
Loading

So, the mappings map to different elements, but the effective difference in distance is tiny. I think this happens when a point is located on the edge between two triangles.

Here I print out the polation elements and weights and then compute the projection point for the old and new boost.
It is possible that the order in which the elements for the nearest-projection are considered changed, leading to different elements to be considered first.

Given that our input functions are spacially dependent, this shouldn't lead to such a big difference in the output. Or could the different points impact the result this extremely in the output of ASTE?

97 different mappings

Diff for 43 [0.031755  0.0892496 0.0909088] -> [0.031755  0.0892496 0.0909088] diff 6.938893903907228e-18
Diff for 54 [0.0910679 0.0459595 0.0272727] -> [0.0910679 0.0459595 0.0272727] diff 0.0
Diff for 74 [0.0512694 0.0345959 0.272726 ] -> [0.0512694 0.0345959 0.272726 ] diff 6.938893903907228e-18
Diff for 331 [0.03890349 0.0902521  0.01557049] -> [0.03890349 0.0902521  0.01557049] diff 0.0
Diff for 340 [0.05321172 0.09000761 0.07610572] -> [0.05321172 0.09000761 0.07610572] diff 0.0
Diff for 341 [0.0532904  0.09000816 0.01482465] -> [0.0532904  0.09000816 0.01482465] diff 1.734723475976807e-18
Diff for 357 [0.02618996 0.08602949 0.00877795] -> [0.02618996 0.08602949 0.00877795] diff 0.0
Diff for 358 [0.06350063 0.08637626 0.07989305] -> [0.06350063 0.08637626 0.07989305] diff 1.962615573354719e-17
Diff for 359 [0.06378877 0.0873704  0.01118595] -> [0.06378877 0.0873704  0.01118595] diff 0.0
Diff for 360 [0.00912367 0.07106188 0.03752179] -> [0.00912367 0.07106188 0.03752179] diff 0.0
Diff for 365 [0.08341745 0.07099416 0.01118875] -> [0.08341745 0.07099416 0.01118875] diff 1.3985787785349405e-17
Diff for 383 [0.01993756 0.08184277 0.00871922] -> [0.01993756 0.08184277 0.00871922] diff 3.469446951953614e-18
Diff for 391 [0.0463409  0.09088561 0.02071189] -> [0.0463409  0.09088561 0.02071189] diff 0.0
Diff for 405 [0.07786282 0.07664384 0.00748544] -> [0.07786282 0.07664384 0.00748544] diff 0.0
Diff for 406 [0.00172516 0.05427121 0.00847953] -> [0.00172516 0.05427121 0.00847953] diff 1.734723475976807e-18
Diff for 409 [0.08979725 0.05430727 0.00847512] -> [0.08979725 0.05430727 0.00847512] diff 0.0
Diff for 415 [0.08683562 0.06205872 0.02275029] -> [0.08683562 0.06205872 0.02275029] diff 1.4304896245381992e-17
Diff for 419 [0.07130562 0.08264604 0.07449454] -> [0.07130562 0.08264604 0.07449454] diff 1.3877787807814457e-17
Diff for 420 [0.07107596 0.08264124 0.01653888] -> [0.07107596 0.08264124 0.01653888] diff 0.0
Diff for 423 [0.02314756 0.08389745 0.01666558] -> [0.02314756 0.08389745 0.01666558] diff 4.9065389333867974e-18
Diff for 440 [0.0584609  0.08732944 0.07103828] -> [0.0584609  0.08732944 0.07103828] diff 1.3877787807814457e-17
Diff for 444 [0.08691084 0.06366636 0.00781267] -> [0.08691084 0.06366636 0.00781267] diff 1.97026712507607e-17
Diff for 567 [0.01831766 0.04207775 0.86050046] -> [0.01831766 0.04207775 0.86050046] diff 0.0
Diff for 623 [0.00988937 0.03617199 0.28819549] -> [0.00988937 0.03617199 0.28819549] diff 0.0
Diff for 773 [0.00671197 0.04212451 0.5846063 ] -> [0.00671197 0.04212451 0.5846063 ] diff 0.0
Diff for 853 [0.0589451  0.03582975 0.43449593] -> [0.0589451  0.03582975 0.43449593] diff 0.0
Diff for 1260 [0.00614159 0.04420755 0.73325251] -> [0.00614159 0.04420755 0.73325251] diff 0.0
Diff for 1590 [0.0148691  0.04964951 0.41585379] -> [0.0148691  0.04964951 0.41585379] diff 6.938893903907228e-18
Diff for 1714 [0.0116469  0.04728232 0.79225595] -> [0.0116469  0.04728232 0.79225595] diff 0.0
Diff for 1753 [0.00753731 0.04997131 0.58459527] -> [0.00753731 0.04997131 0.58459527] diff 0.0
Diff for 1760 [0.00611012 0.04863093 0.44757775] -> [0.00611012 0.04863093 0.44757775] diff 0.0
Diff for 1826 [0.02168015 0.05346873 0.35561084] -> [0.02168015 0.05346873 0.35561084] diff 0.0
Diff for 1897 [0.00747925 0.04780511 0.67721435] -> [0.00747925 0.04780511 0.67721435] diff 0.0
Diff for 1951 [0.00548251 0.04833122 0.41518463] -> [0.00548251 0.04833122 0.41518463] diff 8.673617379884035e-19
Diff for 2165 [0.02437463 0.0504875  0.41859023] -> [0.02437463 0.0504875  0.41859023] diff 0.0
Diff for 2241 [0.00656444 0.04773148 0.73337781] -> [0.00656444 0.04773148 0.73337781] diff 0.0
Diff for 2298 [0.0043105  0.04778281 0.50679067] -> [0.0043105  0.04778281 0.50679067] diff 8.673617379884035e-19
Diff for 2424 [0.02789976 0.06928756 0.15753992] -> [0.02789976 0.06928756 0.15753992] diff 3.122502256758253e-17
Diff for 2455 [0.0099326  0.06322157 0.14285575] -> [0.0099326  0.06322157 0.14285575] diff 1.734723475976807e-18
Diff for 2460 [0.06792024 0.05329357 0.25234925] -> [0.06792024 0.05329357 0.25234925] diff 0.0
Diff for 2478 [0.04280791 0.08663861 0.10738089] -> [0.04280791 0.08663861 0.10738089] diff 0.0
Diff for 2515 [0.14210568 0.04705542 0.1785666 ] -> [0.14210568 0.04705542 0.1785666 ] diff 0.0
Diff for 2522 [0.02131924 0.07008853 0.14618993] -> [0.02131924 0.07008853 0.14618993] diff 3.1031676915590914e-17
Diff for 2546 [0.0922402  0.04834609 0.26126209] -> [0.0922402  0.04834609 0.26126209] diff 1.5515838457795457e-17
Diff for 2604 [0.13817501 0.04109845 0.26333141] -> [0.13817501 0.04109845 0.26333141] diff 0.0
Diff for 2618 [0.02061306 0.07121113 0.13596627] -> [0.02061306 0.07121113 0.13596627] diff 0.0
Diff for 2648 [0.11134667 0.04712509 0.25437413] -> [0.11134667 0.04712509 0.25437413] diff 0.0
Diff for 2667 [0.02919213 0.07456837 0.14235567] -> [0.02919213 0.07456837 0.14235567] diff 0.0
Diff for 2673 [0.00390707 0.05121262 0.19047989] -> [0.00390707 0.05121262 0.19047989] diff 0.0
Diff for 2681 [0.08225485 0.07050358 0.11610616] -> [0.08225485 0.07050358 0.11610616] diff 1.962615573354719e-17
Diff for 2704 [0.02819699 0.07165435 0.15035807] -> [0.02819699 0.07165435 0.15035807] diff 0.0
Diff for 2706 [0.04762203 0.07250641 0.16250567] -> [0.04762203 0.07250641 0.16250567] diff 0.0
Diff for 2725 [0.02080786 0.06706447 0.15519056] -> [0.02080786 0.06706447 0.15519056] diff 0.0
Diff for 2739 [0.00342845 0.05127787 0.17228183] -> [0.00342845 0.05127787 0.17228183] diff 2.7755575615628914e-17
Diff for 2764 [0.11696258 0.04694698 0.15010603] -> [0.11696258 0.04694698 0.15010603] diff 6.938893903907228e-18
Diff for 2776 [0.03372152 0.07182028 0.15720216] -> [0.03372152 0.07182028 0.15720216] diff 2.8609792490763984e-17
Diff for 2777 [0.03253129 0.07940229 0.12751047] -> [0.03253129 0.07940229 0.12751047] diff 3.179800655392251e-17
Diff for 2870 [0.10657692 0.02933997 0.15113578] -> [0.10657692 0.02933997 0.15113578] diff 0.0
Diff for 2888 [0.06086046 0.0191452  0.18185132] -> [0.06086046 0.0191452  0.18185132] diff 2.7755575615628914e-17
Diff for 2899 [0.14752535 0.03398834 0.18289833] -> [0.14752535 0.03398834 0.18289833] diff 2.7755575615628914e-17
Diff for 2900 [0.05343157 0.03284744 0.25893433] -> [0.05343157 0.03284744 0.25893433] diff 0.0
Diff for 2930 [0.09212792 0.03524083 0.11615036] -> [0.09212792 0.03524083 0.11615036] diff 6.938893903907228e-18
Diff for 2938 [0.00547898 0.04060849 0.22066274] -> [0.00547898 0.04060849 0.22066274] diff 0.0
Diff for 2942 [0.06868673 0.03298752 0.26151751] -> [0.06868673 0.03298752 0.26151751] diff 1.3877787807814457e-17
Diff for 2991 [0.10587044 0.02380839 0.15849539] -> [0.10587044 0.02380839 0.15849539] diff 3.469446951953614e-18
Diff for 3003 [0.00264032 0.03751523 0.1117432 ] -> [0.00264032 0.03751523 0.1117432 ] diff 1.3877787807814457e-17
Diff for 3011 [0.13837023 0.03729602 0.17047996] -> [0.13837023 0.03729602 0.17047996] diff 2.8609792490763984e-17
Diff for 3014 [0.08961514 0.03844787 0.09909619] -> [0.08961514 0.03844787 0.09909619] diff 2.0816681711721685e-17
Diff for 3019 [0.06818599 0.01939514 0.18490653] -> [0.06818599 0.01939514 0.18490653] diff 0.0
Diff for 3028 [0.06066809 0.02128966 0.19048013] -> [0.06066809 0.02128966 0.19048013] diff 6.938893903907228e-18
Diff for 3041 [0.05706328 0.03357908 0.2646014 ] -> [0.05706328 0.03357908 0.2646014 ] diff 0.0
Diff for 3059 [0.05130423 0.01880416 0.17701843] -> [0.05130423 0.01880416 0.17701843] diff 3.469446951953614e-18
Diff for 3069 [0.07140464 0.03188875 0.25384761] -> [0.07140464 0.03188875 0.25384761] diff 0.0
Diff for 3083 [0.07462764 0.0123213  0.13290129] -> [0.07462764 0.0123213  0.13290129] diff 0.0
Diff for 3102 [0.14636833 0.03747274 0.17517573] -> [0.14636833 0.03747274 0.17517573] diff 0.0
Diff for 3107 [0.10217736 0.02724643 0.14549859] -> [0.10217736 0.02724643 0.14549859] diff 0.0
Diff for 3110 [0.1128545  0.02856487 0.15885248] -> [0.1128545  0.02856487 0.15885248] diff 3.469446951953614e-18
Diff for 3133 [0.09154206 0.03900084 0.10636968] -> [0.09154206 0.03900084 0.10636968] diff 1.3877787807814457e-17
Diff for 3176 [0.08549466 0.02016327 0.12577194] -> [0.08549466 0.02016327 0.12577194] diff 3.122502256758253e-17
Diff for 3199 [0.05964699 0.03253331 0.25730115] -> [0.05964699 0.03253331 0.25730115] diff 0.0
Diff for 3227 [0.05229692 0.0017486  0.01556087] -> [0.05229692 0.0017486  0.01556087] diff 2.168404344971009e-19
Diff for 3230 [0.00276889 0.03210614 0.0149353 ] -> [0.00276889 0.03210614 0.0149353 ] diff 7.152448122690996e-18
Diff for 3232 [0.08845822 0.03210614 0.0149353 ] -> [0.08845822 0.03210614 0.0149353 ] diff 6.938893903907228e-18
Diff for 3233 [0.07684784 0.01360584 0.01551274] -> [0.07684784 0.01360584 0.01551274] diff 0.0
Diff for 3235 [0.01437927 0.01360584 0.01551274] -> [0.01437927 0.01360584 0.01551274] diff 0.0
Diff for 3238 [0.0380434  0.00199351 0.01477338] -> [0.0380434  0.00199351 0.01477338] diff 7.165583942981724e-18
Diff for 3253 [0.06507855 0.00581193 0.0087868 ] -> [0.06507855 0.00581193 0.0087868 ] diff 0.0
Diff for 3256 [0.02772154 0.00547702 0.01103672] -> [0.02772154 0.00547702 0.01103672] diff 3.576224061345498e-18
Diff for 3260 [0.08276953 0.02140194 0.01110925] -> [0.08276953 0.02140194 0.01110925] diff 1.3877787807814457e-17
Diff for 3261 [0.00845757 0.02140195 0.01110925] -> [0.00845757 0.02140195 0.01110925] diff 3.469446951953614e-18
Diff for 3279 [0.07130201 0.01005897 0.00871969] -> [0.07130201 0.01005897 0.00871969] diff 0.0
Diff for 3287 [0.0448815  0.00100162 0.02070964] -> [0.0448815  0.00100162 0.02070964] diff 0.0
Diff for 3298 [0.08903192 0.03881409 0.05091484] -> [0.08903192 0.03881409 0.05091484] diff 0.0
Diff for 3303 [0.08950909 0.03768465 0.00847584] -> [0.08950909 0.03768465 0.00847584] diff 0.0
Diff for 3305 [0.00171801 0.03768464 0.00847584] -> [0.00171801 0.03768464 0.00847584] diff 6.938893903907228e-18
Diff for 3307 [0.05537713 0.00205687 0.00817447] -> [0.05537713 0.00205687 0.00817447] diff 0.0
Diff for 3371 [0.05090186 0.00199714 0.02223512] -> [0.05090186 0.00199714 0.02223512] diff 0.0

@fsimonis fsimonis mentioned this pull request Jan 27, 2025
Merged
7 tasks
@fsimonis fsimonis force-pushed the add-mapping-tester-compare-script branch from cd5a0b8 to cf248be Compare January 27, 2025 15:01
This was referenced Jan 27, 2025
Open
Open
@fsimonis
Copy link
Member Author

I was able to trace this down to the NP mapping itself projecting to worse elements on macOS-latest.
So, I disabled a test for now on macOS, opened a bug report in the preCICE repository precice/precice#2185, and a PR to add the test again #223,

@fsimonis
Copy link
Member Author

Merging now and rebasing the WIP RPs

@fsimonis fsimonis merged commit 356eaa6 into develop Jan 28, 2025
7 checks passed
@fsimonis fsimonis deleted the add-mapping-tester-compare-script branch January 28, 2025 08:53
@fsimonis fsimonis mentioned this pull request Jan 28, 2025
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@davidscn davidscn Awaiting requested review from davidscn

Assignees

@fsimonis fsimonis

Labels
dependencies Pull requests that update a dependency file documentation Changes affecting the project's documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Test mapping-tester output in CI
2 participants
@fsimonis @davidscn